Driving assistance system, driving assistance method, and program

ABSTRACT

Warning information that facilitates grasping an obstacle located in rear of a vehicle is provided. A driving assistance system including: a first detection section that detects an obstacle from a plurality of captured images of a rear side of a driver of a vehicle; a second detection section that detects the obstacle by analyzing a reflected wave of a measurement wave transmitted to the rear side; and an information output section that determines a direction in which the obstacle is located with respect to the vehicle on the basis of detection results of the first detection section and the second detection section, and that outputs warning information indicating the direction.

TECHNICAL FIELD

The present invention relates to a driving assistance system, a drivingassistance method, and a program.

BACKGROUND ART

Conventionally, there has been known a technique for detecting an imageof a pedestrian from a captured image of a front of a vehicle,superimposing the detected image on the captured image, and displaying aresultant image so that a driver who is driving the vehicle can easilygrasp the presence of the pedestrian (refer to, for example, PatentLiterature 1). As for a rear of the vehicle in addition to the front ofthe vehicle, there has been similarly known a technique for detecting amoving object from a captured image of the rear of the vehicle anddisplaying the captured image of the rear of the vehicle in which themoving object is demarcated by a border (refer to, for example, PatentLiterature 2).

CITATION LIST Patent Literature Patent Literature 1: JP 2010-93610 APatent Literature 2: JP 2009-23565 A SUMMARY OF INVENTION TechnicalProblem

The captured image of the rear of the vehicle is displayed when thedriver looks at a rearview mirror, the captured image is switched overto a captured image of a right rear side of the vehicle when the driverlooks at a right side mirror, and the captured image is switched over toa captured image of a left rear side of the vehicle when the driverlooks at a left side mirror. That is, without moving a line of sight,the driver is unable to obtain a field of view from each mirror and tonotice the moving object in the captured image of any of the mirrorsthat the driver is not looking at. Furthermore, in a case where thedriver is not aware of which mirror the driver has looked at, it isdifficult to grasp, from the captured image, in which direction themoving object in the captured image is located on a rear side of thevehicle.

An object of the present invention is to provide warning informationthat facilitates grasping an obstacle located in rear of a vehicle.

Solution to Problem

The invention set forth in claim 1 is a driving assistance systemincluding:

a first detection section that detects an obstacle from a plurality ofcaptured images of a rear side of a driver of a vehicle;

a second detection section that detects an obstacle by analyzing areflected wave of a measurement wave transmitted to the rear side; and

an information output section that determines a direction in which theobstacle is located with respect to the vehicle on the basis ofdetection results of the first detection section and the seconddetection section, and that outputs warning information indicating thedirection.

According to the invention set forth in claim 1, the obstacle located onthe rear side is detected not only from the plurality of captured imagesof the rear side of the vehicle but also from a measurement result usingthe measurement wave. Therefore, it is possible to detect not only anobstacle at a short detection distance but also an obstacle at a longdetection distance, and warn the driver of the obstacles. Furthermore,the warning information indicating the direction in which the obstacleis located is output on the basis of the detection results. Therefore,it is possible to provide the warning information that facilitatesgrasping the direction in which the obstacle is located in the rear ofthe vehicle difficult for the driver to visually recognize duringdriving. Therefore, it is possible to avoid a collision with an obstacleand assist safety driving.

The invention set forth in claim 2 is the driving assistance systemaccording to claim 1, wherein

the information output section determines a degree of risk of theobstacle on the basis of each of the detection results, and generatesthe warning information when the degree of risk exceeds a threshold.

According to the invention set forth in claim 2, it is possible toprovide the warning information in a case where the degree of risk ishigh to exceed the threshold, and to reduce provision of unnecessarywarning information.

The invention set forth in claim 3 is the driving assistance systemaccording to claim 2, including

a third detection section that detects traffic lanes around the vehiclefrom the captured image, wherein

the information output section determines the degree of risk on thebasis of each detection result of the obstacle and a detection result ofthe traffic lanes.

According to the invention set forth in claim 3, it is possible todetermine the degree of risk by a positional relationship between theobstacle and each traffic lane, and to warn of a possibility ofdeviation from a traveling lane, a collision at a time of a traffic lanechange, and the like.

The invention set forth in claim 4 is the driving assistance systemaccording to claim 2 or 3, wherein

the information output section acquires driving information about thevehicle, and determines the degree of risk on the basis of each of thedetection results and the driving information.

According to the invention set forth in claim 4, it is possible todetermine the degree of risk depending on the obstacle and a drivingsituation of the vehicle, and to warn of possibilities of a collisionwhen the vehicle turns left, a rear-end collision from behind, and thelike.

The invention set forth in claim 5 is the driving assistance systemaccording to any one of claims 2 to 4, wherein

the information output section determines an output mode of the warninginformation depending on the degree of risk.

According to the invention set forth in claim 5, the driver can easilygrasp the degree of risk from a difference in the output mode.

The invention set forth in claim 6 is the driving assistance systemaccording to any one of claims 1 to 5, wherein

the plurality of captured images contain captured images in a pluralityof directions of the rear side, and

the information output section generates one display image by arrangingthe captured images in each of the directions, superimposes the warninginformation indicating the direction in which the obstacle is located onthe captured images in each of the directions, and outputs the onedisplay image to a display section of the vehicle.

According to the invention set forth in claim 6, the driver can grasp asituation of the rear side at a glance. Furthermore, the driver caneasily grasp the direction in which the obstacle is located on the rearside.

The invention set forth in claim 7 is a driving assistance methodexecuted by a driving assistance system, the driving assistance methodincluding:

detecting an obstacle from a plurality of captured images of a rear sideof a driver of a vehicle;

detecting an obstacle by analyzing a reflected wave of a measurementwave transmitted to the rear side; and

determining a direction in which the obstacle is located with respect tothe vehicle on the basis of detection results about the obstacle, andoutputting warning information indicating the direction.

According to the invention set forth in claim 7, the obstacle located onthe rear side is detected not only from the plurality of captured imagesof the rear side of the vehicle but also from a measurement result usingthe measurement wave. Therefore, it is possible to detect not only anobstacle at a short detection distance but also an obstacle at a longdetection distance, and warn the driver of the obstacles. Furthermore,the warning information indicating the direction in which the obstacleis located is output on the basis of the detection results. Therefore,it is possible to provide the warning information that facilitatesgrasping the direction in which the obstacle is located in the rear ofthe vehicle difficult for the driver to visually recognize duringdriving. Therefore, it is possible to avoid a collision with an obstacleand assist safety driving.

The invention set forth in claim 8 is a program for causing a computerto execute:

detecting an obstacle from a plurality of captured images of a rear sideof a driver of a vehicle;

detecting an obstacle by analyzing a reflected wave of a measurementwave transmitted to the rear side; and

determining a direction in which the obstacle is located with respect tothe vehicle on the basis of detection results about the obstacle, andoutputting warning information indicating the direction.

According to the invention set forth in claim 8, the obstacle located onthe rear side is detected not only from the plurality of captured imagesof the rear side of the vehicle but also from a measurement result usingthe measurement wave. Therefore, it is possible to detect not only anobstacle at a short detection distance but also an obstacle at a longdetection distance, and warn the driver of the obstacles. Furthermore,the warning information indicating the direction in which the obstacleis located is output on the basis of the detection results. Therefore,it is possible to provide the warning information that facilitatesgrasping the direction in which the obstacle is located in the rear ofthe vehicle difficult for the driver to visually recognize duringdriving. Therefore, it is possible to avoid a collision with an obstacleand assist safety driving.

Advantageous Effects of Invention

According to the present invention, it is possible to provide warninginformation that facilitates grasping an obstacle located in rear of avehicle.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of configurations of adriving assistance system in the present embodiment.

FIG. 2 is a diagram illustrating an example of detection ranges ofcameras, an ultrasonic sonar, and a millimeter-wave radar.

FIG. 3 is a flowchart of driving assistance processing.

FIG. 4 is a diagram illustrating an example of a composite image.

FIG. 5 is a flowchart of processing for determining a degree of risk.

FIG. 6A is a view illustrating an example of a display screen in a casewhere warning information is not output.

FIG. 6B is a view illustrating an example of a display screen in a casewhere warning information is output.

FIG. 7 is a screen view illustrating another example of the displayscreen in the case where warning information is output.

DESCRIPTION OF EMBODIMENTS

Embodiments of a driving assistance system, a driving assistance method,and a program according to the present invention will be describedhereinafter with reference to the drawings.

FIG. 1 illustrates an example of configurations of a driving assistancesystem 100 according to an embodiment of the present invention.

The driving assistance system 100 according to the present embodiment ismounted in a vehicle and assists a driver of the vehicle in driving byproviding information to the driver.

As illustrated in FIG. 1, the driving assistance system 100 includesthree cameras 1 a to 1 c, a signal processing section 2, an electroniccontrol unit (ECU) 3, a display section 4, and an audio output section5. The driving assistance system 100 also includes a measurement section6 and a driving information output section 7. The measurement section 6and the driving information output section 7 are connected to the ECU 3via a network N such as a controller area network (CAN).

The cameras 1 a to 1 c continuously capture images of a rear side of thedriver and sequentially generate time-series captured images. Thecameras 1 a to 1 c are provided at positions including, for example, aposition near a license plate and positions of left and right sidemirrors. The captured images are subjected to signal processing by thesignal processing section 2 and then output to the ECU 3.

(ECU)

The ECU 3 detects an obstacle such as another vehicle or a pedestrianlocated on a rear side of the driver, generates warning informationagainst the detected obstacle, and outputs the warning information tothe display section 4 or the audio output section 5.

The ECU 3 includes a first detection section 31, a second detectionsection 32, a third detection section 33, an information output section34, and a storage section 35.

The first detection section 31 detects an obstacle from each of thecaptured images generated by the cameras 1 a to 1 c.

The second detection section 32 analyzes a reflected wave of ameasurement wave transmitted to the rear side of the vehicle by themeasurement section 6. The second detection section 32 detects anobstacle located on the rear side on the basis of an analysis result.

The third detection section 33 detects traffic lanes around the vehiclefrom each of the captured images generated by the cameras 1 a to 1 c.

The information output section 34 determines a direction in which theobstacle is located with respect to the vehicle on the basis ofdetection results of the first detection section 31 and the seconddetection section 32. The information output section 34 outputs warninginformation indicating the determined direction.

The storage section 35 can store the captured images generated by thecameras 1 a to 1 c, a measurement result of the measurement section 6,driving information output from the driving information output section7, and the like. As the storage section 35, a storage medium such as aflash memory or an EEPROM can be used.

It is noted that functions of the first detection section 31, the seconddetection section 32, the third detection section 33, and theinformation output section 34 can be realized by processing performed byhardware such as IC chips, or can be realized by software processing inwhich a processor such as a CPU executes a program. In a case ofrealizing the functions by the software processing, the program can bestored in the storage section 35.

The display section 4 may be either an in-vehicle monitor having ageneral aspect ratio or a rearview mirror type monitor having a ratio atwhich a width is larger than a height. The display section 4 can displaythe captured images generated by the cameras 1 a to 1 c in accordancewith an instruction from the ECU 3. Furthermore, the display section 4can display a warning message, a warning mark, and the like on the basisof the warning information output from the ECU 3.

The audio output section 5 is a speaker or the like that outputs a soundin accordance with an instruction from the ECU 3. The audio outputsection 5 can output a warning sound on the basis of the warninginformation output from the ECU 3.

The measurement section 6 transmits a measurement wave such as a soundwave or a radio wave toward the rear side of the vehicle, receives areflected wave of the measurement wave, and outputs the reflected waveto the ECU 3 as a measurement result of measuring the rear side. Thedriving assistance system 100 according to the present embodimentincludes an ultrasonic sonar 6 a and a millimeter-wave radar 6 b as themeasurement section 6. The ultrasonic sonar 6 a transmits an ultrasonicwave and receives a reflected wave of the ultrasonic wave. Themillimeter-wave radar 6 b transmits a radio wave called a millimeterwave having a radio wave wavelength of approximately 1 to 10 mm (afrequency band of 30 to 300 GHz,) and receives a reflected wave of theradio wave. It is noted that a sensor using a measurement wave otherthan the ultrasonic wave and the millimeter wave may be provided as themeasurement section 6.

The driving information output section 7 detects a driving state of thevehicle and outputs a detection result to the ECU 3 as drivinginformation. The driving assistance system 100 according to the presentembodiment includes a vehicle speed sensor 7 a and a blinker sensor 7 bas the driving information output section 7. The vehicle speed sensor 7a detects a rotation speed of an axle of the vehicle. The ECU 3 canobtain a traveling speed of the vehicle from the detected rotation speedof the axle. The blinker sensor 7 b detects a driver's operation on aright or left blinker. The ECU 3 can detect a start of a traffic lanechange of the vehicle from a detection result of the blinker sensor 7 b.It is noted that other sensors such as a steering angle sensor or thelike may be provided as the driving information output section 7 as longas the other sensor is capable of detecting the driving state.

FIG. 2 illustrates an example of obstacle detection ranges of thecameras 1 a to 1 c, the ultrasonic sonar 6 a, and the millimeter-waveradar 6 b.

As illustrated in FIG. 2, the camera 1 a is provided near a rear licenseplate of a vehicle D1. A detection range 51 a of the camera 1 a islocated in rear of the vehicle D1 similarly to a detection range of arearview mirror, and has an angle of view equal to or greater than anangle of view of the rearview mirror. On the other hand, the cameras 1 band 1 c are provided near the left and right side mirrors, respectively.Detection ranges 51 b and 51 c of the cameras 1 b and 1 c are located ona rear left side and a rear right side of a driver M1, respectively, andeach have an angle of view equal to or greater than an angle of view ofthe left or right side mirror.

The ultrasonic sonar 6 a is provided near the left side mirror. Adetection range 56 a of the ultrasonic sonar 6 a is located in a rangefrom a left side to the rear left side of the vehicle D1, which is ablind spot of the camera 1 b. The detection range 56 a enables detectionof a pedestrian M2 located in the blind spot on the rear left side.

The millimeter-wave radar 6 b is provided in rear of the vehicle D1similarly to the camera 1 a. A detection range 56 b of themillimeter-wave radar 6 b is located in rear of the vehicle D1 similarlyto the detection range 51 a of the camera 1 a. However, a detectabledistance d2 of the detection range 56 b in rear of the vehicle D1 islarger than a distance d1 of the detection range 51 a. It is, therefore,possible to detect an obstacle D2 located farther than the detectionrange 51 a of the camera 1 a. For example, the distance d1 isapproximately 50 m, while the distance d2 is approximately 110 m.

It is noted that the obstacle detection ranges described above are givenas an example and the present invention is not limited to the detectionranges. Positions, directions, angles of view, detection distances, andthe like of the detection ranges can be arbitrarily determined dependingon laws and regulations and a custom of a state where the vehicletravels, cases of accidents prone to occur in the state, and the like.

FIG. 3 is a flowchart illustrating driving assistance processingexecuted by the ECU 3. This processing is repeatedly performed duringdriving.

As illustrated in FIG. 3, in the ECU 3, the first detection section 31detects an obstacle from a plurality of captured images generated by thecameras 1 a to 1 c (Step S1). Examples of the obstacle as an object tobe detected include a person, an animal, another vehicle, a guardrail,and a building. The first detection section 31 can obtain informationrelated to the obstacle such as a classification of, for example,whether the obstacle is a vehicle or a person, a distance from thevehicle to the obstacle, and a relative speed of the obstacle withrespect to the vehicle by analyzing the captured images.

While a detection method by the first detection section 31 is notlimited to a specific one, a detection method using, for example,machine learning or artificial intelligence can be used. Detection canbe performed using histograms of oriented gradients (HOGs) as themachine learning or using deep learning as the artificial intelligence.In a case of using the machine learning or the deep learning, the firstdetection section 31 learns features of an image of an object to bedetected by using, as images for learning. Examples of the image includecaptured images that are obtained by capturing the object to be detectedsuch as a vehicle or a person from various directions and that differ invehicle type or perspective. The first detection section 31 then detectsimage regions that coincide with the features of the image of the objectto be detected learned from the captured images, respectively.

Next, the third detection section 33 detects traffic lanes around thevehicle from each of the captured images (Step S2). For example, thethird detection section 33 detects straight lines by Houghtransformation after binarizing the captured images. Then, the thirddetection section 33 detects straight lines, for example, each toward avanishing point from among the straight lines as the traffic lanes, ordetects straight lines each detected at substantially the same positionover a plurality of frames as the traffic lanes. It is noted that atraffic lane detection method is not limited to this method, and apublicly-known detection method can be used. For example, the thirddetection section 33 may detect traffic lanes by detecting a guardrail,a median strip, or the like without limiting to detection of road signssuch as white lines or yellow lines that demarcates the traffic lanes.

The detection processing in Steps S1 and S2 is preferably performed onone composite image obtained by combining three captured images.

FIG. 4 illustrates an example of a composite image 11 of three capturedimages 11 a, 11 b, and 11 c.

Since one module can perform detection processing on one composite image11, configurations can be made simple and a cost can be reduced in acase of one composite image, compared with a case where three modulesperform processing on the three captured images 11 a to 11 c inparallel, respectively. Furthermore, processing time can be made shortin the case of processing on the one composite image 11, compared with acase where one module performs processing on the three captured images11 a to 11 c in order.

On the other hand, the second detection section 32 detects an obstacleby analyzing the measurement result of the measurement section 6 (StepS3). At this time, the second detection section 32 can obtaininformation related to the obstacle such as the distance from thevehicle to the obstacle, an azimuth and the relative speed of theobstacle with respect to the vehicle from the measurement result.

When no obstacle is detected by the first detection section 31 and thesecond detection section 32 (Step S4: NO), this processing is ended.When an obstacle is detected (Step S4: YES), the information outputsection 34 determines a direction in which the obstacle is located withrespect to the vehicle on the basis of the detection results (Step S5).

The information output section 34 can determine the direction in whichthe obstacle is located depending on in which detection range theobstacle is detected. When the obstacle is detected by, for example, thecaptured image by the camera 1 a or the measurement result of themillimeter-wave radar 6 b, the information output section 34 candetermine that the direction in which the obstacle is located is adirection of the detection range of the camera 1 a or themillimeter-wave radar 6 b, that is, a direction of the rear of thevehicle. Furthermore, when the obstacle is detected by the capturedimage by the camera 1 b or the camera 1 c, the information outputsection 34 can determine that the direction in which the obstacle islocated is a direction of the rear left side of the vehicle that is inthe detection range of the camera 1 b or of the rear right side of thevehicle that is in the detection range of the camera 1 c. When theobstacle is detected by the ultrasonic sonar 6 a, the information outputsection 34 can determine that the direction in which the obstacle islocated is a direction of the blind spot from the left side to the rearleft side of the vehicle, which is the detection range of the ultrasonicsonar 6 a.

Since the detection range 51 a of the camera 1 a partially overlaps thedetection ranges 51 b and 51 c of the cameras 1 b and 1 c, the sameobstacle is often detected from the captured images of the cameras 1 ato 1 c. Whether the obstacles are the same can be determined by, forexample, whether the distances from the vehicle are the same. In thiscase, the direction may be determined on the assumption that theobstacle is detected from the captured image in which the obstacle islarger in size. It is thereby possible to provide warning informationthat facilitates grasping the direction in which the obstacle islocated.

Next, the information output section 34 determines a degree of risk onthe basis of the detection results about the obstacle (Step S6). Theinformation output section 34 can determine the degree of risk on thebasis of at least one of the detection result about the traffic lanesand the driving information about the vehicle acquired from the drivinginformation output section 7 together with the detection results aboutthe obstacle. Determination accuracy for the degree of risk improves bya combination of the detection results about the obstacle with thedetection result about the traffic lanes or the driving information. Thedegree of risk may be either a binary index value representing whetherrisk is high or low, or a multi-index value representing a level of riskin a plurality of levels.

FIG. 5 is a flowchart illustrating procedures for processing fordetermining the degree of risk. This processing for determining thedegree of risk is executed per detected obstacle.

As illustrated in FIG. 5, when the direction in which the detectedobstacle is located is a rearward direction (Step S11: YES), theinformation output section 34 calculates a time until the vehiclecollides with the obstacle (Time-To-Collision (TTC)) (Step S12). Theinformation output section 34 can calculate the TTC from the informationrelated to the obstacle and obtained by analyzing the captured images orthe measurement result of the millimeter-wave radar 6 b. Examples of theinformation include a position of the obstacle, the distance between thevehicle and the obstacle, and the relative speed of the obstacle withrespect to the vehicle. The information output section 34 may calculatethe TTC by further combining the traveling speed of the vehicle obtainedfrom the vehicle speed sensor 7 a.

When the calculated TTC is shorter than a threshold (Step S13: YES), theinformation output section 34 determines that there is a possibility ofa collision with the obstacle located in rear of the vehicle, that is, arear-end collision, and that the degree of risk is high (Step S14). Theinformation output section 34 can also determine the degree of risk at aplurality of levels in such a manner that the degree of risk is higherin level as the TTC is shorter by comparing the TTC with a plurality ofthresholds.

When the TTC is not shorter than the threshold (Step S13: NO), theinformation output section 34 determines that the degree of risk of therear-end collision is low (Step S21).

On the other hand, when the direction in which the obstacle is locatedis a rear left side direction or a rear right side direction (Steps S11:NO and S15: YES), the information output section 34 determines whetherthe blinker sensor 7 b detects the operation on the left or rightblinker in the direction in which the obstacle is located (Step S16).When the blinker sensor 7 b detects the operation on the correspondingblinker (step S16: YES), the information output section 34 determinesthat there is a possibility of a collision with the obstacle travellingin a traffic lane in a direction in which the vehicle is about to make atraffic lane change and located on the rear side of the vehicle, andthat the degree of risk is high (Step S14). At this time, theinformation output section 34 may determine the degree of risk at aplurality of levels in such a manner that the degree of risk is higherin level as the distance between the vehicle and the obstacle is smallerby comparing the distance with a plurality of thresholds.

When the blinker sensor 7 b does not detect the operation on the left orright blinker (Step S16: NO), the information output section 34determines whether a distance between the vehicle and the traffic laneadjacent to the vehicle among the traffic lanes detected by the thirddetection section 33 is smaller than a threshold (Step S17). When thedistance is smaller than the threshold (Step S17: YES), the informationoutput section 34 determines that there is a possibility that thevehicle deviates from the traffic lane in which the vehicle is travelingand enters the adjacent traffic lane in which the obstacle is located onthe rear side, and that the degree of risk is high (Step S14). Theinformation output section 34 may determine the degree of risk at aplurality of levels in such a manner that the degree of risk is higherin level as the distance between the vehicle and the traffic lane issmaller by comparing the distance with a plurality of thresholds.

When the blinker sensor 7 b does not detect the operation on the left orright blinker and the distance between the vehicle and the adjacenttraffic lane is not smaller than the threshold (Steps S16: NO and S17:NO), the information output section 34 determines that the degree ofrisk of the traffic lane change or a traffic lane deviation is low (StepS21).

When the direction in which the obstacle is located is a direction ofthe blind spot from the left side to the rear left side (Steps S11: NO,S15: NO, and S18: YES), the information output section 34 determineswhether the blinker sensor 7 b detects the operation on the left blinker(Step S19). When the blinker sensor 7 b detects the operation on theleft blinker (Step S19: YES), the information output section 34determines that there is a possibility of a collision with the obstaclewhen the vehicle turns left and that the degree of risk is high (StepS14). At this time, the information output section 34 may determine thedegree of risk at a plurality of levels in such a manner that the degreeof risk is higher in level as the distance between the vehicle and theobstacle is smaller by comparing the distance with a plurality ofthresholds. Furthermore, the information output section 34 may determinethat the degree of risk is high not only when the distance is smallerbut also when the traveling speed detected by the vehicle speed sensor 7a is higher than a threshold.

When the blinker sensor 7 b does not detect the operation on the left orright blinker (Step S19: NO), the information output section 34determines whether the distance between the vehicle and the obstacleobtained by analyzing the captured image or the measurement result ofthe ultrasonic sonar 6 a is smaller than a threshold (Step S20). Whenthe distance to the obstacle is not smaller than the threshold (StepS20: NO), the information output section 34 determines that the degreeof risk of a collision of the vehicle when turning left is low (StepS21).

When the distance to the obstacle is smaller than the threshold (StepS20: YES), the information output section 34 determines that there is apossibility of a collision with the obstacle located in the blind spotand that the degree of risk is high (Step S14). At this time, theinformation output section 34 may determine the degree of risk at aplurality of levels in such a manner that the degree of risk is higherin level as the distance between the vehicle and the obstacle is smallerby comparing the distance with a plurality of thresholds. Furthermore,the information output section 34 may determine that the degree of riskis high not only when the distance is smaller but also when thetraveling speed detected by the vehicle speed sensor 7 a is higher thana threshold.

After determining the degree of risk, the information output section 34determines whether the determined degree of risk exceeds a threshold asillustrated in FIG. 3 (Step S7). When the degree of risk is high andexceeds the threshold (Step S7: YES), the information output section 34generates warning information indicating the direction in which theobstacle is located, and outputs the warning information to the displaysection 4 or the audio output section 5 (Step S8). When the degree ofrisk does not exceed the threshold (Step S7: NO), the information outputsection 34 ends the present processing without generating the warninginformation.

Examples of the warning information include an arrow pointing to thedirection in which the obstacle is located, a marker blinking in thedirection, and a message sentence notifying the driver of the directionwhen the warning information is output by the display section 4.Examples of the warning information include a voice message notifyingthe driver of the direction, a notification sound unique to eachdirection, and a melody sound when the warning information is output bythe audio output section 5. The warning information can contain not onlythe direction in which the obstacle is located but also informationrelated to the obstacle such as the classification (vehicle, person, orthe like) of the obstacle, the relative speed of the obstacle withrespect to the vehicle, the TTC, and a separation distance.

The information output section 34 can determine an output mode of thewarning information depending on the degree of risk. Setting the outputmode in such a manner that a degree of emphasis of the obstacle ishigher as the degree of risk is higher can facilitate driver's graspingthe presence and the direction of the obstacle. In a case where thewarning information is output by the display section 4, for example, asize, a color, a brightness, presence or absence of blinking, and thelike of characters or images to be displayed as the warning informationcan be varied depending on the degree of risk. Furthermore, in a casewhere the warning information is output by the audio output section 5,for example, a pitch of the sound, a tone, a type of the melody sound, acontent of the voice message, and the like can be varied depending onthe degree of risk.

Moreover, the information output section 34 can generate one displayimage in which the captured images in the directions are arranged andoutput the display image to the display section 4. The informationoutput section 34 can output the warning information to the displaysection 4 to be superimposed on the captured image in each direction ofthe display image. The warning information indicates the direction inwhich the obstacle is determined to be located. The one display imagecan facilitate driver's grasping situations in the directions on therear side and particularly the direction in which the obstacle islocated among the directions. Since the display section 4 can confirmthe situations on the rear side, the driver can concentrate a driver'sline of sight on a front side, and safety on the front side can beenhanced.

FIG. 6A illustrates an example of a display screen in a case where thewarning information is not output, and FIG. 6B illustrates an example ofa display screen in a case where warning information is output. In theexamples of the display screen of FIGS. 6A and 6B, a captured image 41 bof the rear left side, a captured image 41 a of the rear, and a capturedimage 41 c of the rear right side are arranged in this order anddisplayed as one display image.

In the example of the display screen illustrated in FIG. 6A, the warninginformation is not displayed since no obstacle is detected or anobstacle is detected but the degree of risk is low. On the other hand,in the example of the display screen illustrated in FIG. 6B, an obstacleD2 is detected in the captured image 41 c of the right rear side. Theobstacle D2 travels in the traffic lane adjacent to a right side of thetraffic lane in which the vehicle is traveling, and the TTC of which issmaller than the threshold. An image 401 of a frame is, therefore,displayed as the warning information. The image 401 of the frame issuperimposed on an outer edge of the captured image 41 c of the rearright side where the obstacle D2 is located. A color of the image 401 ofthe frame is switched over to blue, yellow, and red in order as adistance to the obstacle D2 is smaller and the degree of risk is higherin level.

Furthermore, in the example of the display screen illustrated in FIG.6B, the operation on the blinker is not detected. However, an image 402of a blinking line is displayed to be superimposed on a white line ofthe right traffic lane as the warning information since it is determinedthat the distance between the right traffic lane in which the obstacleD2 is located and the vehicle is small and the degree of risk is high.The image 402 of the line can draw driver's attention to the righttraffic lane. The white line of the right traffic lane is detected fromboth the captured images 41 a and 41 c. However, the image 402 of theline is displayed in the captured image 41 a larger in a size of thewhite line, so that the image 402 can draw the driver's attention moreeasily. It is noted that an image of a line covering the traffic laneper se may be superimposed on the traffic lane as the warninginformation as an alternative to the white line. To facilitate callingattention, it is preferable to superimpose the image of the line on thecaptured image 41 c larger in the size of the traffic lane.

FIG. 7 illustrates another example of the display screen in the casewhere the warning information is output. In the example of the displayscreen illustrated in FIG. 7, a captured image 42 b of the rear leftside, a captured image 42 a of the rear, and a captured image 42 c ofthe rear right side are arranged in this order and displayed as onedisplay image.

In the example of the screen illustrated in FIG. 7, the obstacle D2having the TTC smaller than the threshold is detected in the capturedimage 42 a of the rear. An image 403 of a frame is, therefore,superimposed on an outer edge of the captured image 42 a as one piece ofthe warning information. Furthermore, as one piece of the warninginformation, a message sentence 404 is displayed. The message sentence404 warns the driver that the obstacle D2 is another vehicle, which islocated in rear of the subject vehicle, and the relative speed of theobstacle D2 is 5 km/h or higher. In parallel with the display, the audiooutput section 5 also outputs an audio 405 warning the driver that thevehicle as the obstacle D2 is approaching at the relative speed of 5km/h or higher.

As described so far, the driving assistance system 100 according to thepresent embodiment detects the obstacle located in rear side of thevehicle on the basis of not only the plurality of captured images of therear of the vehicle but also the measurement result using themeasurement wave. It is possible to detect and warn of not only anobstacle at a short detection distance but also an obstacle at a longdetection distance. Furthermore, according to the present embodiment,the direction in which the obstacle is located is determined on thebasis of the detection results, and the warning information indicatingthe direction in which the obstacle is located is output. It is therebypossible to provide the warning information that facilitates driver'sgrasping the direction in which the obstacle is located in the rear ofthe vehicle difficult for the driver to visually recognize duringdriving, and to avoid a collision with the obstacle and assist safetydriving.

The embodiment described above is an appropriate example of the presentinvention, and the present invention is not limited to the embodiment.Modifications can be made as appropriate without departing from the gistof the present invention.

For example, the display section 4 is not necessarily one monitor andmay be three monitors corresponding to the cameras 1 a to 1 c,respectively. It is possible to perform display as illustrated in FIGS.6A, 6B, and 7 by arranging the monitors adjacently to allow the monitorsto display the captured images of the cameras 1 a to 1 c, respectively.In this case, the warning information may be output to the monitordisplaying the captured image in which the obstacle is detected sincethe three monitors correspond to the rear left side, the rear side, andthe rear right side of the vehicle, respectively. The driver can grasp astate of the rear side at a glance by the arranged monitors, and caneasily grasp in which direction the obstacle is located by the monitorto which the warning information is output.

The present application claims priority based on Japanese PatentApplication No. 2019-102550 filed on May 31, 2019, the entire contentsof which are incorporated herein.

REFERENCE SIGNS LIST

100 Driving assistance system

1 a to 1 c Camera

3 ECU

31 First detection section

32 Second detection section

33 Third detection section

34 Information output section

35 Storage section

4 Display section

5 Audio output section

6 a Ultrasonic sonar

6 b Millimeter-wave radar

7 a Vehicle speed sensor

7 b Blinker sensor

1. A driving assistance system comprising: a first detector that detectsan obstacle from a plurality of captured images of a rear side of adriver of a vehicle; a second detector that detects an obstacle byanalyzing a reflected wave of a measurement wave transmitted to the rearside; and an information outputter that determines a direction in whichthe obstacle is located with respect to the vehicle on the basis ofdetection results of the first detector and the second detector, andthat outputs warning information indicating the direction.
 2. Thedriving assistance system according to claim 1, wherein the informationoutputter determines a degree of risk of the obstacle on the basis ofeach of the detection results, and generates the warning informationwhen the degree of risk exceeds a threshold.
 3. The driving assistancesystem according to claim 2, comprising a third detector that detectstraffic lanes around the vehicle from the captured images, wherein theinformation outputter determines the degree of risk on the basis of eachdetection result of the obstacle and a detection result of the trafficlanes.
 4. The driving assistance system according to claim 2, whereinthe information outputter acquires driving information about thevehicle, and determines the degree of risk on the basis of each of thedetection results and the driving information.
 5. The driving assistancesystem according to claim 2, wherein the information outputterdetermines an output mode of the warning information depending on thedegree of risk.
 6. The driving assistance system according to claim 1,wherein the plurality of captured images contain captured images in aplurality of directions of the rear side, and the information outputtergenerates one display image by arranging the captured images in each ofthe directions, superimposes the warning information indicating thedirection in which the obstacle is located on the captured images ineach of the directions, and outputs the one display image to a displayof the vehicle.
 7. A driving assistance method executed by a drivingassistance system, the driving assistance method comprising: detectingan obstacle from a plurality of captured images of a rear side of adriver of a vehicle; detecting an obstacle by analyzing a reflected waveof a measurement wave transmitted to the rear side; and determining adirection in which the obstacle is located with respect to the vehicleon the basis of detection results about the obstacle, and outputtingwarning information indicating the direction.
 8. A non-transitorycomputer-readable medium storing a program for causing a computer toexecute: detecting an obstacle from a plurality of captured images of arear side of a driver of a vehicle; detecting an obstacle by analyzing areflected wave of a measurement wave transmitted to the rear side; anddetermining a direction in which the obstacle is located with respect tothe vehicle on the basis of detection results about the obstacle, andoutputting warning information indicating the direction.